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(54) Node to node interrupt mechanism in a multi-processor system 



(57) An interrupt mechanism handles an interrupt 
transaction between a source processor and a target 
processor on separate nodes in a multi-processor sys- 
tem. The nodes are connected to a network through 
node interface controls between the node and the net- 
work. The transaction begins by initiating the interrupt 
transaction at the source processor. The interrupt mech- 
anism detects if the target processor is at a remote node 
on a system bus across the network, and if it is the mech- 
anism sends an ignore signal to the source processor. 



Then the mechanism suspends the interrupt transaction 
at the source processor if it detects the target processor 
is at a remote node. The mechanism performs an ACK/ 
NACK (acknowledge/non-acknowiedge) operation at 
the target processor and returning an ACK signal or a 
NACK signal to the source processor across the net- 
work. This ACK/NACK signal wakes-upthe source proc- 
essor. The source processor sends interrupt data to the 
target processor if an ACK signal is received and aborts 
the interrupt transaction if a NACK signal is received. 
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Description 

This invention relates to interrupt mechanisms in 
multiple processor systems and to such an interrupt 
mechanism where the interrupt from one processor is 
being sent to another processor on a different system 
bus. 

In multiple processor systems where the multiple 
processors are on the same system bus, interrupts be- 
tween processors are exchanged with little or no delay. 
An interrupt from a first processor to a second processor 
is either acknowledged (ACK signal) or not acknowl- 
edged (NACK signal) by the second processor. Depend- 
ing upon whether an ACK or a NACK is received by the 
first processor as status back from the second proces- 
sor, the first processor proceeds to send the interrupt 
data or aborts the interrupt transaction. 

When a multiple processors system has multiple 
system busses, the various system busses are connect- 
ed through a global interconnect such as a bus or com- 
munication network. In this environment there can be a 
significant delay between the initiating processor send- 
ing out the interrupt address to begin the interrupt trans- 
action and the initiating processor receiving the ACK or 
NACK status back from a target processor on another 
system bus. 

If interrupt transaction requires too much time to 
complete, the processor initiating the interrupt transac- 
tion may either waste processing time handling the in- 
terrupt transaction, re-initiate the transaction, or abort 
the transaction. 

Particularand preferred aspects of the invention are 
set out in the accompanying independent and depend- 
ent claims. Features of the dependent claims may be 
combined with those of the independent claims as ap- 
propriate and in combinations other than those explicitly 
set out in the claims. 

In an embodiment of this invention, the above prob- 
lem is solved by performing an interrupt transaction in 
a multi-processor system between a source processor 
on a local system bus and a remote target processor on 
a remote system bus. The local system bus is connected 
through a local node interface control to a global inter- 
connect, and the target system bus is connected 
through a remote node interface control to the global in- 
terconnect. The source processor initiates an interrupt 
request on the local system bus. The local node inter- 
face control captures the interrupt request and sends an 
ignore status to the source processor if the interrupt re- 
quest is addressed to a target processor on the remote 
system bus. Also in this event, the local node interface 
forwards the interrupt request across the global inter- 
connect to the remote node interface control. The 
source processor suspending its interrupt transaction 
process in response to the ignore status. The remote 
node interface control responds to the interrupt request 
to send the request to the target processor. The target 
processor responds to the interrupt request and returns 



an ACK (acknowledge) signal if the target processor is 
ready to process the interrupt transaction and returns a 
NACK (non-acknowledge) signal if the target processor 
is busy. The remote node interface control returns the 
5 ACK or NACK signal across the global interconnect to 
the local node interface control. The local node interface 
control responds to the ACK signal or NACK signal to 
send the ACK or NACK signal to the source processor. 
The source processor responds to the ACK signal or the 
10 NACK signal for waking-up the interrupt transaction 
process in the source processor and sends an interrupt 
data packet to the target processor in response to the 
ACK signal and aborts the interrupt transaction in re- 
sponse to a NACK signal. 
J5 Preferably, the local node interface control has a de- 
tect module responsive to the target address to detect 
the target processor is on a remote system bus. A send 
module at the local node interface responds to the de- 
tect module detecting the target processor is on a re- 
20 mote system bus and sends the ignore status to the 
source processor and forwards the interrupt address to 
the remote node interface control. 

Preferably, the detect module responds to the target 
address to detect the target processor is on the local 
25 system bus. The target processor sends an ACK signal 
or a NACK signal over the local system bus to the sou rce 
processor so that the interrupt transaction is handled on 
the local system bus. 

Preferably, the local node interface control has a re- 
30 issue module responsive to an ACK signal or a NACK 
signal from the target processor via the remote node in- 
terface control and reissues the interrupt request with 
the source address back to the source processor. An 
assert module in the local node interface control re- 
35 sponds to the ACK signal from the target processor to 
assert an ACK signal onto the local system bus, and re- 
sponds to the NACK signal from the target processor for 
asserting a NACK signal onto the local system bus. 
The source processor has a wake-up module re- 
40 sponsive to the interrupt request and wakes-up the in- 
terrupt transaction process in the source processor. A 
send module in the source processor responds to the 
ACK signal and sends the interrupt data packet to the 
target processor. An abort module in the source proc- 
45 essor responds to the NACK signal and aborts the in- 
terrupt transaction. 

Preferably, the local node interface control re- 
sponds to the interrupt data packet and sends the Inter- 
rupt data packet to the remote node interface control 
50 along with an interrupt transaction completion message. 
The remote node interface control responds to the in- 
terrupt data packet and sends the interrupt data packet 
to the target processor. Also the remote node interface 
control responds to the interrupt transaction completion 
55 message and frees the resources of the remote node 
interface control. 

Preferably, the local node interface control re- 
sponds to the NACK signal and sends an interrupt trans- 
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action completion message to the remote node interface 
control. The remote node interface control responds to 
the interrupt transaction completion message and frees 
the resources of the remote node interface control. 

The great advantage and utility of the present in- £ 
vention is ability to handle interrupt transactions with a 
remote target processor without unduly burdening the 
source processor. 

Exemplary embodiments of the invention are de- 
scribed hereinafter, by way of example only, with refer- n 
ence to the accompanying drawings, in which: 

Fig. 1 shows the preferred embodiment of the inter- 
rupt mechanism for handling interrupt transactions on 
the same system bus or across an interconnect to an- 
other system bus. n 

M Fig. 2 illustrates an example of the operating envi- 
ronment for the present invention. 

Fig. 3, composed of Figs. 3A, 3B and 3C, illustrates 
the operations performed by a source processor, a first 
node interface control, a second node interface control 2C 
and a target processor to accomplish the interrupt trans- 
action across a global interconnect in accordance with 
a preferred embodiment of the invention. 

The operating environment for the invention is illus- 
trated in Fig. 1. Fig. 1 shows a multi-processor system 25 
having a global interconnect connecting three nodes in 
a multi-processor system having multiple processors at 
each node. The multiple processors at each node are 
connected by a system bus. For example processor 32 
and processor 34 are interconnected by system bus 0 30 
(SB 0 ). The system bus, in turn, is connected to the glo- 
bal interconnect 30 by node interface control 0. Global 
interconnect 30 could be as simple as a cross-bar switch 
or a bus or it could be a communication network. Each 
of the three nodes in Fig. 1 follows this same structure 35 
with multiple processors connected to a system bus, 
their system bus connected to a node interface control, 
and the node interface controls connected across the 
global interconnect. 

Each of the processors in the multi-processor sys- 40 
tern has cache memory and shares main memory with 
other processors in the multi-processor system. For pur- 
poses of this embodiment, the registers used in each 
processor, as shown for processor 32 (P 0 ), are an inter- 
rupt dispatch register 36, interrupt dispatch status reg- 45 
ister 38, interrupt receive register 40 and an interrupt 
receive status register 42. These registers are shown 
only for processor 32 (Processor P 0 ) but are present in 
all of the processors in the multi-processor system of 
Fig. 1 . The node interface control devices 44, 46 and 48 so 
are all intelligent devices fabricated from ASICs (Appli- 
cation Specific Integrated Circuits). 

Fig. 2 illustrates the preferred embodiment of the 
invention where an interrupt transaction initiated by a 
first processor is handled either as a local interrupt ss 
transaction on the same system bus, or as a remote in- 
terrupt transaction through a global interconnect to a tar- 
get processor on a second system bus. The operations 



begin by the source processor P 0 initiating the interrupt 
transaction (INTR TRAX) in operation 10. With the 
transaction on the local system bus, operation 12 de- 
tects whether the interrupt transaction is for a processor 
on the local system bus or for a processor on a remote 
system bus. If the interrupt transaction is for a local proc- 
essor, the operation flow branches "Yes" to the local in- 
terrupt transaction operations 14. If the interrupt trans- 
action is for a second processor across a communica- 
tion network to a second system bus, the operation flow 
branches "No" from decision 12 to suspend operation 
16. Suspend operation 16 suspends the interrupt trans- 
action at P 0 , the initiating processor for the interrupt 
transaction. While the transaction is suspended at initi- 
ating processor P 0 , remote acknowledge/no-acknowl- 
edge (NACK) operations 18 are performed. These re- 
mote ACK/NACK operations involve communicating an 
interrupt request to a target processor through the com- 
munication network and a second system bus to which 
the target processor is attached. The target processor 
will return an ACK/NACK status signal depending on 
whether it is busy. 

Operation 20 detects whether the return status is 
ACK or NACK. If the target processor acknowledges the 
interrupt request, then the operation flow branches to 
operation 22 where the initiating processor P 0 sends to 
the target processor P 1 on the second bus the interrupt 
transaction data. Operations 22 complete the transmis- 
sion of interrupt data between initiating processor P 0 
and target processor P 1 . 

If the decision operation 20 detects that the status 
return from the target processor is NACK, then the op- 
eration flow branches to abort operation 24. In the abort 
operation 24, the initiating processor aborts the interrupt 
transaction. 

Fig. 3, composed of Figs. 3A, 3B and 3C, illustrates 
the operations performed by an initiating processor, P 0 , 
a target processor P-,, a node interface control 0 at the 
node containing the initiating processor and a node in- 
terface control 1 at the node containing the target proc- 
essor. Fig. 3 is organized in four columns indicating the 
operations performed at the initiating processor P 0 , the 
node interface control 0 on the same system bus with 
the initiating processor, the node control interface con- 
trol 1 on the same system bus with the target processor, 
and the target processor 1 . 

In Fig. 3A the operation begins at the initiating proc- 
essor P 0 where operation 50 initiates an interrupt trans- 
action (INTR TRAX) addressed to target processor P v 
Operation 52 at P 0 drives the interrupt address (INTR 
ADDR) as dispatched from the interrupt dispatch regis- 
ter on to system bus 0 (SB 0 ). Node interface control 0 
(NIC 0 ) in operation 54 then captures the interrupt ad- 
dress (IT ADDR). Operation 54 also detects that this in- 
terrupt address is for a processor P-, located at node 1 . 

In response to the detection of the interrupt address 
for another node, operation 56 sends an IGNORE status 
out on the SB 0 . Processor P 0 in operation 58 captures 



3 



BNSOOCID: <EP 0817038A2_I_> 



5 



EP 0 817 038 A2 



6 



the IGNORE status as the reply to P 0 driving out the 
interrupt address. In response to the IGNORE status, 
P 0 suspends the interrupt transaction in operation 60. 
Meanwhile the node interface control 0 (NIC 0 ) in oper- 
ation 62 sends an interrupt request to NIC V As shown 
in Fig. 1 NIC, is at node 1 , which contains multiple proc- 
essors including the target processor P, interconnected 
by a system bus SB^ 

At NIC 1 , operation 64 receives the interrupt request 
for target processor P v NIC^ then drives the interrupt 
address contained in the interrupt request on system 
bus SB-, during operation 66. 

Target processor captures the interrupt address 
from the system bus and identifies itself as the recipient 
of the interrupt transaction. Decision operation 70 at tar- 
get processor P-, tests whether the processor is BUSY, 
i.e handling an interrupt transaction from another proc- 
essor. If the processor is BUSY, the operation flow 
branches from decision operation 70 to send operation 
72 in Fig. 3C. If the target processor is NOT BUSY, then 
the operation flow branches to send operation 74. Op- 
eration 74 sends an acknowledge (ACK) out on SB-,. 
The node interface control 1 captures this acknowledge 
status in its operation 76. The remainder of the operation 
flow in response to an ACK status is shown in Fig. 3B. 

After NIC, captures the ACK status from the target 
processor, operation 76 in Fig. 3B sends an ACK reply 
to NIC 0 . At NIC 0 operation 78 receives the ACK reply. 
In response to the ACK reply NIC 0 in operation 80 re- 
issues the interrupt address on SB 0 . At the initiating 
processor P 0 , operation 82 captures the interrupt ad- 
dress and wakes up the interrupt transaction process in 
processor P 0 . 

After NIC 0 reissues the interrupt address, operation 
84 at NIC 0 asserts the ACK status for the target proc- 
essor on SB 0 . Initiating processor P 0 being awakened 
in operation 82, captures the ACK status in operation 
86. Operation 88 in the initiating processor then sends 
the interrupt data packet out on system bus SB 0 . 

NIC 0 at operation 90 captures the interrupt data 
packet. Operation 92 then sends the interrupt data pack- 
et along with a completion message to NIC-,. 

At NICt operation 94 receives the interrupt data 
packet and the completion message. Nl^ in operation 
96 sends the interrupt data out on SB V After the inter- 
rupt data is sent out on SB-, , operation 98 frees the re- 
sources at NIC-, to handle the next transaction. 

At the target processor P-, operation 100 captures 
the interrupt data put on system bus SB 1 by NIC-,. After 
the interrupt data packet is captured by the target proc- 
essor P 1( the interrupt transaction initiated by initiating 
processor P 0 is complete. 

Fig. 3C illustrates the operation flow in the event 
that the target processor P., is BUSY, and operation 72 
in Fig. 3C has sent a NACK status out on SB, . Operation 
102 in NIC, captures the NACK status on system bus 
SB V In response to the NACK status, operation 104 
sends a NACK reply to NIC 0 . 



Operation 106 in NIC 0 receives the NACK reply and 
initiates operation 108. Operation 108 reissues the in- 
terrupt address on SB 0 . At the initiating processor P 0 , 
operation 110 captures the interrupt address and in re- 

5 sponse thereto wakes up the interrupt transaction 
processing at the initiating processor P 0 . NIC 0 , after re- 
issuing the interrupt address, in operation 112 asserts 
the NACK status on SB 0 . This is the NACK status that 
originated from target processor P-,. Initiating processor 

10 p Q jn operation 114 captures this NACK status. Then 
processor P 0 aborts the interrupt transaction in opera- 
tion 116. After the interrupt transaction is aborted, initi- 
ating processor P 0 sets an interrupt transaction retry to 
initiate the interrupt transaction at a later time. 

is After sending the NACK status, NIC 0 in operation 
120 sends a completion message to NIC-,. Operation 
120 in effect generates the response after NIC 0 has as- 
serted the NACK status on SB 0 . This response is simply 
a completion message. There is no interrupt data packet 

20 sent by operation 120. 

At NIC-, the completion message is received by op- 
eration 122. NIC-,, in response to the completion mes- 
sage, initiates operation 124. Operation 124 frees all of 
the resources allotted to the interrupt transaction in 

25 NIC V NIC! is then ready to process the next interrupt 
transaction. 

While the invention has been particularly shown 
and described with reference to a preferred embodiment 
thereof, it will be understood by those skilled in the art 
30 that various other changes in the form and details may 
be made therein without departing from the scope of the 
invention. 



1. A method for handling an interrupt transaction be- 
tween a source processor and a target processor 
on separate nodes in a multi-processor system, the 
40 nodes being connected to a network through net- 
work interface controls between the node and the 
network, said method comprising the computer im- 
plemented steps of: 

4S . initiating the interrupt transaction at the source 

processor; 

detecting the target processor is at a remote 
node on a system bus across the network; 

so 

suspending the interrupt transaction at the 
source processor if said detecting step detects 
the target processor is at a remote node; 

55 performing an ACK/NACK (acknowledge/non- 

acknowfedge) operation at the target processor 
and returning an ACK signal or a NACK signal 
to the source processor across the network; 
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waking up the source processor when an ACK 
signal or NACK signal is received from the tar- 
get processor; and 

sending the interrupt data from the source proc- s 
essor to the target processor if an ACK signal 
is received and aborting-the interrupt transac- 
tion if a NACK signal is received. 

2. The method of claim 1 further comprising the com- io 
puter implemented steps of: 

detecting the target processor is at a local node 
on the same system bus as the source proces- 
sor; is 

performing a local interrupt transaction if said 
detecting step detects the target processor is 
at a local node. 

20 

3. The method of claim 1 wherein said initiating step 
sends an interrupt address identifying the target 
processor as the target of the interrupt transaction. 

4. The method of claim 3 wherein said suspending 2S 
step comprises the computer implemented steps of: 

sending an ignore signal to the source proces- 
sor in response to said detecting step detecting 
the target processor is at a remote node; 30 

capturing the ignore signal at the source proc- 
essor and in response thereto suspending the 
interrupt transaction at the source processor. 

35 

5. The method of claim 4 wherein said waking-up step 
comprises the computer implemented steps of: 

sending an ACK signal or NACK signal to the 
source processor from the target processor; 40 



quest on the local system bus; 

said local node interface control capturing the 
interrupt request and sending an ignore status 
to said source processor if the interrupt request 
is addressed to a target processor on the re- 
mote system bus and forwarding the interrupt 
request across the gbbal interconnect to the re- 
mote node interface control; 

said source processor suspending its interrupt 
transaction process in response to the ignore 
status; 

said remote node interface control responsive 
to the interrupt request to send the request to 
the target processor; 

said target processor responsive to the inter- 
rupt request returning an ACK (acknowledge) 
signal if the target processor is ready to process 
the interrupt transaction and returning a NACK 
(non-acknowledge) signal if the target proces- 
sor is busy; 

said remote node interface control returning the 
ACK or NACK signal across the global intercon- 
nect to the local node interface control; 

said local node interface control responsive to 
the ACK signal or NACK signal to send the ACK 
or NACK signal to the source processor; 

said source processor responsive to the ACK 
signal or the NACK signal for waking-up the in- 
terrupt transaction process and sending an in- 
terrupt data packet to the target processor in 
response to the ACK signal and aborting the in- 
terrupt transaction in response to a NACK sig- 
nal. 



capturing the ACK signal or NACK signal at the 
source processor and in response thereto wak- 
ing-up the interrupt transaction at the source 
processor. 45 

6. Interrupt mechanism apparatus for performing an 
interrupt transaction in a multi-processor system 
between a source processor on a local system bus 
and a remote target processor on a remote system so 
bus. the local system bus being connected through 
a local node interface control to a global intercon- 
nect and the target system bus being connected 
through a remote node interface control to the glo- 
bal interconnect, said interrupt mechanism appara- ss 
tus comprising: 

said source processor initiating an interrupt re- 



7. The apparatus of claim 6 wherein said interrupt re- 
quest includes a target address identifying the tar- 
get processor and said local node interface control 
comprises: 

a detect module responsive to the target ad- 
dress and detecting the target processor is on 
a remote system bus; 

a send module responsive to the detect module 
detecting the target processor is on a remote 
system bus and sending an ignore status to the 
source processor and forwarding the interrupt 
address to the remote note interface control. 

8. The apparatus of claim 7 and in addition: 
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said detect module responsive to the target ad- 
dress and detecting the target processor is on 
the local system bus; and 

said target processor sending an ACK signal or s 
a NACK signal over the local system bus to said 
source processor. 

9. The apparatus of claim 7 wherein said local node 
interface control further comprises: u> 



said local node interface control responsive to 
the NACK signal and sending an interrupt transac- 
tion completion message to the remote node inter- 
face control. 

14. The apparatus of claim 13 wherein: 

said remote node interface control responsive 
to the interrupt transaction completion message 
and freeing the resources of the remote node inter- 
face control. 



reissue module responsive to an ACK signal or 
a NACK signal from the target processor via the 
remote node interface control reissuing the in- 
terrupt request with the source address back to 15 
the source processor; 



an assert module responsive to the ACK signal 
from the target processor for asserting an ACK 
signal onto the local system bus; and 



said assert module responsive to the NACK 
signal from the target processor for asserting a 
NACK signal onto the local system bus. 

1 0. The apparatus of claim 9 wherein said source proc- 
essor comprises: 



a wake-up module responsive to the interrupt 
request and waking-up the interrupt transaction 30 
process in said source processor; 

a send module responsive to the ACK signal for 
sending the interrupt data packet to the target 
processor; 35 

an abort module responsive to the NACK signal 
aborting the interrupt transaction. 



11. The apparatus of claim 10 wherein: 40 
said local node interface control responsive to 
the interrupt data packet and sending the interrupt 
data packet to the remote node interface control 
along with an interrupt transaction completion mes- 
sage. 45 



12. The apparatus of claim 11 wherein: 



said remote node interface control responsive 
to the interrupt data packet and sending the in- so 
terrupt data packet to the target processor; and 



said remote node interface control responsive 
to the interrupt transaction completion mes- 
sage and freeing the resources of the remote ss 
node interface control. 



13. The apparatus of claim 9 wherein: 
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(54) Node to node interrupt mechanism in a multi-processor system 



(57) An interrupt mechanism handles an interrupt 
transaction between a source processor and a target 
processor on separate nodes in a multi-processor sys- 
tem. The nodes are connected to a network through 
node interface controls between the node and the net- 
work. The transaction begins by initiating the interrupt 
transaction at the source processor. The interrupt mech- 
anism detects if the target processor is at a remote node 
on a system bus across the network, and if it is the mech- 
anism sends an ignore signal to the source processor. 
Then the mechanism suspends the interrupt transaction 
at the source processor if it detects the target processor 
is at a remote node. The mechanism performs an ACK/ 
NACK (acknowledge/non-acknowledge) operation at 
the target processor and returning an ACK signal or a 
NACK signal to the source processor across the net- 
work. This ACK/NACK signal wakes-up the source proc- 
essor. The source processor sends interrupt data to the 
target processor if an ACK signal is received and aborts 
the interrupt transaction if a NACK signal is received. 
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